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The Effectiveness of an Electronic Performance Support Systems on 
Learning and Performance. 



Theo J. Bastiaens 
Wim J. Nijhof 
HarmenJ. Abma 

University ofTwente, The Netherlands 



This research is measuring the effectiveness of Electronic Performance 
Support Systems (EPSS). Some of the major advantages of EPSS, like 
increasement of productivity and better learning outcomes are evaluated with 
insurance agents working on laptops. Theoretical statements, research 
framework and hypothesis are presented. The conclusions are drawn and no 
significant improvement on productivity or learner results has been seen. 



In the last ten years the Computer Based Training (CBT) market grew tremendously and the 
quality of the programs increased. Sound and vision were added, educational technology 
improved the design and structure. Nowadays key words are just>in time learning and learning by 
doing. For that CBT had to be revised and integrated in the workplace. One way to integrate 
learning in the workplace is using Electronic Performance Support Systems (EPSS). EPSS 
supports and monitors employees while they are woridng; but EPSS is not only an alternative for 
the human master in the master/joumeyman paradigm, it is more. EPSS is an integration of all 
the electronical sources and tools employees need to do their job. Developers and researchers 
working in the field of EPSS have hi^ expectations. EPSS will improve productivity and 
learning outcomes. What are their assumptions? A short historical overview of corporate training 
can help. 

Since the early seventies most of the employees in business and industry are trained by 
classroom instruction. They are treated as a homogeneous group. The master who's the expert 
teaches the desired knowledge and skills. The problems of classroom instruction are related to 
costs and benefits. There is a lot of money involved; turnover costs etc. The benefits of classroom 
instruction are relatively low. Employees have to learn skills in a theoretical way . Learners have 
problems to relate the classroom instruction to their own specific situation. Reasons to change the 
paradigm to the good old master/joumeyman paradigm. Since the middle ages this paradigm has 
been proven. At the workplace a master is doing his job. The journeyman observes the master 
and tries to imitate the him. A cheap and efficient way. Nowadays it is hard to use this paradigm 
because of the number of employees. But there are some good properties in this training on the 
job. Related to classroom instmction training on the job is often cheap. Emplc^ees don't have to 
leave their workplace anymore (Jacobs & Me Griffin, 1987, Jacobs, 1990). The learning 
processes are more concrete and active (MOnch, 1990). The transfer is higher because of the 
identical situation between the learning environment and the workplace (Wexl^ & l^tham, 
1991). All these advantages stay when CBT is used in the woikplace. Besides that people can get 
trained whenever and where they want. But (TBT has problems too. Employees have to stop 
working when they use CBT. Training with CBT is still an separate event (Geiy 1989). 
Employees will benefit firom the integration of learning with their job. EPSS is possibly the 
solution for all the problems stated above. This research will try to e;q>lore some conclusions. 

EPSS is a concept which integrates the electronical sources emplc^ees need to do their 
job. Employees need tools to perform, they have to look up information quick to use in their job, 
they want to learn certain subject-matters or skills when needed and they want to get expert 
advise to guide them through difficult parts of their job. For that reason an EPSS environment 
exists of four components (fig 1). 




© copyright, Bastiaens, Nijhof, & Abma, 1995 
13-1 



3 



Figure 1 . The components of an EPSS. 




EPSS provides employees with just*in time knowledge, information and l earning at the right 
moment (Gery 1992). High transfer, no need to leave the workplace and more active learning 
processes are advantages of training on the job. Specific EPSS advantages are immediate access 
to information and learning, a reduction of the amount of formal training in advance of lagif 
performance and a reduction of the supervisor time spend on emplc^ees. It has the aHHitinnal 
advantage that the responsibility focus changes fi'om the trainer and t raining program to 
individual job specific learning experiences (Gery, 1992). Performance support systems can be 
important in employees self management or self-directed teams (Bramer & Senbatta, 1993) and 
will improve the workers productivity (Raybould, 1990, 1991). Law (1994) tries to find scientific 
evidence for the surplus value of EPSS from current cognitive theory. He relates performance 
limitations to cognitive explanations. Performance requiring large amounts of factual knowledge, 
proficiency of skills infrequently utilized on the job or simultaneous processing of a large amnnnt 
of information have an cognitive explanation. It places high demands on long term memory 
without adequate processing, declarative and procedural knowledge that decayed or in a limited 
capacity of the working memory. In Law's opinion EPSS can provide an extension of long term 
memory and reducing the working memory load. 



Problem Statement 

In the first paragraph a lot has been stated but little has been proven. This research project 
attempts to evaluate an EPSS on the effectiveness. In literatiue a few disadvantages were found. 
Just-in time training at the worlqrlace, providing employees small task-oriented tr aining granules 
and employees taking control of their own learning process will create problems. Clark (1992) 
gathered some subjects we have to take into account. In her opinion employees may fail to build a 
unified picture of their job when they have to distract information from an EPSS. Several little 
information parts will create a fragmented knowledge base. Especially novices need a high level 
overview of the content to relate details of training. She doubts about ^e learner control in EPSS. 
She cites research of Milheim and Martin (1991) which proves that learner control is not as 
effective as instructional control. It is expected that the intrc^uction and implementation of EPSS 
will summon resistance. Employees are not likely to give up working the old way*. And when 
not, there is an other problem. The pressure in their work. En:q)loyees will not have the time to 
engage the training support. A more philosophical question is related to the long term effects of 
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EPSS. Will it 'de*skiir workers? Will it 'de-motivate* workers? Or will it automate the low level 
tasks and bring in more time to perform tasks on higher level (Carr, 1 992)? 

It is not possible to give answers on to all the questions. In general we want to measure the 
effectiveness of EPSS. We expect EPSS to be more effective than the "old situation”. 
"Effectiveness” is split up in effectiveness on learning (learner results) and effectiveness on 
working (productivity). The next section will explain the setting and will further state our 
underlying expectations. 



Research questions 

The project is executed in cooperation with a large Dutch insurance company. This company is 
selling insurance products to their clients by a widespread network of insurance agents. These 
agents are working for the company on a commission based q^m. The company administrates 
the sold products and provides background information and training to their agents. The 
insurance agents operate from their home ofBce 1^ visiting (potential) clients. The training 
program exists of an introductipn course for new agents and specialized (related to specific 
products) courses for advanced agents. In general training means classroom instruction, experts 
provide knowledge, manuals and textbooks to read at home. The training department ascertained 
this way of training not to be (cost)effective. The expensive courses didn't lead to better outcomes 
in learner results and productivity. As the policy for the future was to develop a computer tool to 
communicate better and quicker with their head-office (sending, receiving data), the training 
department joined a project which provides all their agents with laptop computers in the near 
future. The idea was bom to develop an EPSS for insurance-agents. Tlie EPSS was developed 
and exists of information, advise and learning about their products. The agent can read 
information about a product and can automatically calculate with their clients data The EPSS 
will give an advise based on the specific situation of the client The agents support their sales 
presentation with slides and pictures. They also get Conqtuter Based Training (CBT). The 
casd>ased modules enclose all their products. A self test indicates the preparedness of knowledge 
and skills related to the products. The compaiQ^ expects such an electronically environment to be 
more effective. To test the expectations three main groups were compared. The first main group 
is the group working and learning in a traditional way. This group gets a traditional classroom 
instruction and they work pen/paper based (forms, hand-books) and get information from 
manuals. The second main group is woridng and learning with an EPSS. The third group is a 
control group. For several reasons it was not possible to do preliminary research. So we compare 
the two treatments (traditional and EPSS) with a 0-group. Ilus third main group didn't have any 
training and is working on a traditional way. In order to judge about learning and working we 
split up the treatments in two parts; the first is the working part. It's the tool program in the 
electronical environment and the forms and hand-books in the traditional way. The second is the 
learning part; the CBT in the electronical environment and the classroom instruction in the 
traditional group. To make the description of this research more complex we had to deal with 
other variants. In table one the groups are stated. 
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Table 1: Three main groups and their variants. 



TREATMENT 


n 


1 


TOOL & CBT (EPSS) 


12 


2A 


T(X)L & CLASS 


8 


2B 


PAPER & CLASS 


4 


3A 


T(X)L 


5 


3B 


PAPER 


7 



The main hypothesis "EPSS is more effective than traditional classroom training and traditional 
pen/paper based woridng". This thesis is split up in several research questions. The question 
related to training is; Agents in the EPSS group, trained with CBT, will score higher on a 
learners test. In other words these agents will have higher learner outcomes. The research 
question related to working is; agents working with EPSS will score higher on selling. The 
productivity will increase. As this EPSS is a new system, we are also interested in the evaluation 
of the components involved, the feedback, the support and the opinion of the users. Answers and 
questions on these components are important for future development. In this paper we will only 
try to answer the questions above. Although the next section "methodology" will give an 
overview of the whole project, we will not go further into the quality of the developed ^SS. At 
the end we will try to show causes for the outcomes. 



Methodology 

The data collection in this project is split up in a qualitative- and a quantitative part. Table two 
shows the method used to gather the information, the target group, the treatment and the amount 
of persons involved. 

At the start of the project we collected the sales results of 1993. To give an opinion 
about the productivity we used their sales of insurance’s in the field of armuity. These insurance’s 
are topics in the CBT and classroom training. At the end of the project their results over 1994 
were collected. It was expected that agents using an EPSS would sell more armuities compared 
with agents not using an EPSS. The next measurements were the interviews (n= 12). Four 
people of every group were interviewed. They answered on structured questions related to our 
variables (see next chapter). At the end of the interview they got a case. They had to respond to a 
practical situation and had to process the data. This practical situation was observed to give 
insights in people using their computer or manuals and forms. This was done to relate 
performance differences to productivity and learning results. These learner results were measured 
on all the agents (n= 36) in a test. They all got the same test on pension insurance’s. Next all the 
agents got a questionnaire (n=36). The questiormaire was separated in two different parts. The 
first was a general part based on our variables, the second was related to the specific treatment. 
After they filled up the questiormaire the whole group held a discussion (n= 36). This session was 
important to collect the ideas of the agents. They have to work all day with EPSS. They know 
what is effective and what they like. So gathering their experiences is worthwhile. To compare 
the assertions of the agents we talked to their managers (n=8). What is their opinion about EPSS, 
productivity and change? They discussed positions related to EPSS. In the section results and 
conclusions our findings are forwarded. 
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Table 2. overview of the methodology. 



Ol 


XA 


OAl 


03 


04 


OA2 


05 


06 


07 


08 


Ol 


XB 


OBI 


03 


04 


OB2 


05 


06 


07 


08 


Ol 


XC 


OCl 


03 


04 


0C2 


05 


06 


07 


08 


Ol 


XD 


ODl 


03 


04 


OD2 


05 


06 


07 


08 



oi 

Oal.dl 

03 

04 

Oa2..d2 

06 

07 

08 



Sales results *93, agents (n=36) 
Inteview , agents (n=12) 
Observation, agents (12) 
Learning test, agents (n=36) 
(^estionnaire, agents (n=36) 
Discussion, agents (n=36) 
Questionnaire, managers (n=8) 
Sales results *94, agents (n=36) 



XA TOOL&CBT (n= 12) 

XB TOOL&CLASS (n=8) 

XC PAPER&TOOL (n= 5) 

XD PAPER (n= 7) 



The theoretical construct behind EPSS 

A lot has been written about EPSS, less has been proved. From literature a theoretical fiamewodc 
has been constructed. The framework encloses the variables who exert an influence on the EPSS. 
Work, learning, treatment and background have their effect. In table three we state our variables. 
Work, learning and background are general. The treatments T(X)L, CBT CLASS and PAPER 
are specific. 

Table 3. the framewofl^ with an influence on EPSS. 



Constructs 



Variables 



Work 
quality 
productivity 
attitude towards work 
performance 

Learning 
way of learning 

Tool 

communication, 
advising client 
obtaining information 



O 




service, communication, adequate advise, sales talk 
sales results 

satisfaction, self-confidence 
independency, commercial, daily work 

processing, guidance, study conceptions 

inter&oe, support 
advise 

time to learn, applicable, studiousness 
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CBT 

communication 

pedagogical/didactic dimensions 
general 



intei&ce 

structure, e?q)erience validity, feedback cooperative 
learning 

usability, integration with TOOL 



Class 

training and support 
p edagogical/didactic dimensions 

effects 



training, support 

structure, ejq)erience validity, feedback cooperative 

learning 

effects 



Paper 

presentation 

advise 

obtaining information 



presentation, siqiport 
advise, help 

time to learn, applicable, sources 



Background 
personal data 
experience 

attitude towards iiuiovation 



age, sex 

educational background, working and computer 
experience 

working with new technology 



Results 

When an opinion is given about EPSS we combine TOOL and CBT. Traditional woridng and 
learning is PAPER and CLASS. To be able to use the data the independence of the groups had to 
be sure. First this independence had to be tested on age, working erqrerience, educational 
background and iimovation willingness. Oii age and working experience no si gnificant difference 
was found (one-way analysis of variances for age, n= 36, p=.4S one tailed and for working 
experience n= 36, p=.27 one tailed). Also for educational background no significant difference 
was found. But on iimovation willingness the score of the TOOL-group was significantly higher 
regarding the CLASS group (Maim-Whim^ Test, n=9, p=.016 one tailed) and the PAPER group 
(Maim-Whirn^ Test, n=12, p=.037 one tailed). A reason for this result is hard to give. It is 
possible that people in the TOOL groiq> are more affected to the project and more iimovative for 
the reason that th^ are working with new technology. But the other groups working with new 
technology (TOOL&CBT, TOOL&CLASS) aient more iimovative than the traditional groups. 
The conclusion is that there is no difference b^ween the groups excqrt the one above. We take 
the view that the groups are selected at random. 

We had to test the reliability of the items involved. The limit was .60 (Cronbachs 
Alpha). Because of the size of the groups we had some problems with the reliability of the items. 
The variables that were useful (alpha > .60) are shown in table four. 
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Table 4 The variables used 



Variable 


Alpha 


Work 


“ 


Ixaming 


- 


Tool 


inter&ce 


.75 


studiousness 


.63 


CBT 


interlace 


.75 


feedbadr 


.95 


cooperative learning 


.95 


effects 


.63 


Class 


training 


.87 


siqqmrt 


.83 


coqrerative learning 


.76 


effects 


.94 


Pdpa 


presentation 


.83 


support 


.86 


advise 


.70 


help 


.77 


obtaining information 


.73 


Background 


attitude towards innovations 


.73 



Hypothesis 1 related to support With these items we tested the following hypothesis: 
Agents woridng with the computer (TOOL, TOOL&CBT, TOOL&CLASS) like the presentation 
and the interface of the computer more than the agents working pen and paper bas^ (PAPER, 
PAPER&CLASS). This is not the case. However agents do like the presentation of their 
traditional haiulbooks and form more (Man- Whitney Test, n=36, p=.(X)S one tailed). The mean 
rank (15.52 for the computer and 25.27 for the traditional way) shows the advantage for the 
traditional fimn. This was stqrported by results on the interviews. Agents like the traditional 
presentation mme because it is quicker (glancing through a manual) to look things up. Reasons 
for this preference may have to ^ with the construction and userfriendliness of the ^SS or the 
agents are not accustomed to the EPSS yet (they were working with it for four months when the 
evaluation took place). The used hypothesis was that the computer (TOOL&CBT, TOOL) would 
support the agents better while they are working compared with the traditional way 
(PAPERJ*APBt&CLASS). The result is that there is no significant difference. The computer 
didn’t siqiport the agent better or worse (Man- Whitney Test, n= 36, p=.29 one tailed). This result 
is against all esqKCtations. It was expected that the computer would provide a better support than 
the traditional n^ods. 
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Hypothesis 2 related to learning When we looked at the learning part the following 
hypothesis was stated; agents learning with CBT (TOOL&CBT) like the presentation and the 
interface of the CBT more than the agents who followed a classroom training and who had to 
deal with a teacher presentation (TOOL&CLASS, PAPER&CLASS). On the test (Man-Whitn^ 
Test, n=24, p=.012 one tailed) the mean rank, 9.2S for CBT and 1S.7S for the classroom 
instruction shows that the presentation of the teacher was a^reciated more. In the interviews the 
most common reason to like traditional classroom training more was the contact with other 
colleagues. Th^ see training as a social event. In their daily work th^ don’t have contacts with 
colleagues. Th^ like seeing each other at a training. Now th^ are a^d that CBT will take 
them away their social event. The next statement was that agents learning with CBT like to learn 
alone (TOOL&CBT), . Agents in classroom instruction like to learn together with other 
colleagues (TOOL&CLASS, PAPER&CLASS). The answer on this hypothesis is that this is 
right. Agents in the classroom instruction do like to learn in cooperation with others and agents 
learning with CBT do like learning alone (Man-Whitnor Test, n=24, p=.0007 one tailed). The 
last hypothesis related to learning was agents learning with CBT (TOOL&CBT) think that th^ 
learn more effective than agents in classroom training (TOOL&CLASS, PAPER&CLASS). In 
fact agents in classroom training think that th^ learn most effective in this way. The hypothesis 
is not true. CBT is not appreciated as most effective (Man-Whitn^ Test, n=24, p=.0S0 one 
tailed). 

Next we looked at the results on the learning test. The test was split up in a case and in 
questions to test their knowledge. In the practical situation the agents had to give an ideal advise 
to a fictionary client. With this case the quality of the solution is measured. With the questions 
their background knowledge was tested. Both are important in their daily work. When we look at 
the results of the cases and we compare the groups trained with CBT (TOOL&CBT), classroom 
training (TOOL&CLASS, PAPER&CLASS) and no training (TOOL&PAPER) we see that there 
is only a difference between training and no training. Classroom training compared with no 
training shows a significant difference (T-test, n=24, p= .043, separate variance one tailed). CBT 
compared with no training also shows a significant difference (T-test, n=23, p= .018, separate 
variance one tailed). In both cases training was more effective than no training, this is not 
surprising. The difference on the case between CBT and the classroom training was not 
significant (T-test, n= 23, p= .24, separate variance one tailed). Conclusion is that it doesn’t 
matter whether agents were trained with CBT or in a traditional classroom. When we look at the 
questions to test their background knowledge there is no difference between the group with CBT 
and the group with classroom training (T-Test, n=23, p=.149, separate variance one tailed). 
There is a significant difference between classroom training and no training ((T-Test, n=24, 
p=.009, pooled variance one tailed). The score of people that were trained was higher. CBT 
compared with no training didn’t show a significant result ((T-test, n= 23, p= .060, separate 
variance one tailed).. That is remaikable. The standard deviation causes this effect (CBT 
standard deviation = 27.67, no training 41.860). The overall conclusion is that there is no 
significant difference between CBT and the traditional training. 

Hypothec related to work The company is training people to make more mon^ in 
future. The next part will conq)are the sales re^ts between the thi^ groups. We only looked at 
one part of their sales. This results are related to the sold aimuities in 1993 and 1994. Several 
other influences (like a difference in the political situation in the Netherlands) were not 
measured. It is hard to relate these results to the treatment Knowing that these results are not 
only caused by the treatment we carefully state the following: the productivity of the group 
learning with the computer will be significantly higher in 1994 (the year th^ started working 
with the computer). First a comparison between the sales in 1993 and 1994 took place. The 
significant difference between the two years were split up between the groups (CBT, classroom 
training and no training). A one-way analysis of the variance was not sigptiificant (n= 36, p=.230, 
one tailed). The conclusion is that there is no difference in sales between the groups. It doesn’t 
matter whether agents are trained or not, it has no significant effect on the selling of aimuities. 
Finally we test the hypothesis whether agents working with the computer sold more than agents 
working pen/paper based in 1994. The group working with new technology (TOOL&CBT, 



TOOL&CLASS, TOOL) and the group working in a traditional way (PAPER&CLASS, PAPER) 
were compared on their selling results. The expectation is; agents working with new technology 
will sell more in 1994 than agents working in a traditional way. This hypothesis seems not to be 
true (T-Test, n= 36, p=.23, one-way). There is no significant difference between the two. 
Working with the computer will not result in a higher productivity on annuities. Did the agents 
sell anything more at all in 1994 ? Yes, they did, but not all of them. The CBT group didn’t sell 
significantly more (paired T-Test, n=12, p=.0SS). The group trained in a classroom situation did 
have a higher productivity (paired T-Test, n=12, p=.0014). The group without training did also 
do significantly better (paired T-Test, n=36, p=.015). The way people work didn’t have any 
special influence. The group woildng with new technology has hi^er results in 1994 (paired T- 
Test, n=2S, p=.003). The group working in a traditional way scored significantly higher too 
(paired T-test, n=19, p=.0052). The overall conclusion training doesn’t contribute aiQlhing to the 
productivity in this situation. 



Conclusions 

We tried to evaluate the effects of EPSS on learning and productivity. The setting was a large 
insurance company in The Netherlands. This project was a pilot project. This pilot had a lot of 
limiting conditions. But that is not imusual in this type of research. Although we would have 
loved larger groups. We tried to measure EPSS in the field were the surplus value is expected. 
For that a framework was developed. In the fiameworic are variables which have an impact on 
the effectiveness of the EPSS. The expectations in the first chapter were tested. In general the 
EPSS in this project is not so successful as in literature often is stated. This EPSS did have 
information, learning and advise. Although the researchers have some remarks on the 
development of the EPSS, it is not a badly constructed one. Maybe the integration of the CBT can 
be better (those small granules) but this lack of integration is not the reason that agents don’t 
like the EPSS a lot when they have to learn with it Of course there are reasons, like the social 
event reason, not related to the effects. But also on the effects the influence of EPSS is hardly 
measurable. Productivity doesn’t increase. The learners results are almost the same as in the 
traditional situation. An advantage as just-in time learning is hard to see. Agents don’t have the 
time to look up information. They postpone training. As a result the just-in time training 
becomes inadequate working. The employees didn’t take the responsibility for learning . In future 
the direct managers has to take the responsibility for their learning. The advise and information 
part of the EPSS didn’t do any better than the traditional manuals. The agents thought it is 
quicker to look up information in books. The overall conclusion is they like the old way better. 
What are the reasons for it. Is this the expected resistance? No, they told us some good thing s 
They like the automation of certain tasks, like the fill up of forms and counting the benefits for 
their clients. As is stated by Can (1992) they know have the time to do other things, like talking 
to their clients. And that is important. The agents also like it that they now have the possibility to 
show or present something to clients. This was harder in the traditional situation. Now they feel 
more professional and a client is trusting them more. Positive findings of EPSS haven been seen 
in the project. These advantages are on the whole related to controlling the costs. There is no 
direct improvement of productivity and learner benefits. Further research has to be done to 
improve ^SS and prove these findings in other settings. 
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Principles of CBI Design and the Adult Learner: The Need for 
Further Research 



Reid A. Bates 
Diart L. Seyler 
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The increasing use of computer-based instruction in HRD applications is 
accompanied by a growing need for guidelines directing its effective design 
and use. Current research has failed to provide this guidance. Research 
deficiencies include a limited number of studies in many areas, 
contradictory findings, and questionable generalizability to adults in the 
workplace due to an almost exclusive use of students college age and 
younger as research subjects. 



The development of faster, less e>q)ensive, more adaptable and more user-friendly microcomputers which 
can incorporate a wide range of software capabilities such as sound, graphics, and video has made a 
conputer revoluticm in educatic^ and training a dramatic possibility. It has been predicted, for example, 
that, by the end of this century, most of our children's education will take place via computer (Lepper, 
1985) and that 50% or more of all industrial training will involve computers by the year 2000 
(Harrington, McElroy, & Morrow, 1990). Such expectations call for research aimed at identifying 
specific variables related to the effectiveness of computer-based instruction as an instructional 
methodology. 

Ccmq^uter-based instruction (CBI), or the process of instruction in which a computer is used to 
present instructional materials to a learner, has been studied diligently for more than three decades. The 
research suggests that, across content and educational levels, CBI is effective in improving performance 
and attitudes toward instruction as well as accomplishing the same learning objectives as conventional 
instructirai in significantly less tinte (Dalton, Hannah, & Hooper, 1989). However, the research on CBI 
and related variables has been conducted predomiriately in fonnal classroom settings with subjects college 
age or younger. Few studies aimed at assessing the effectiveness of CBI or identifying specific variables 
related to the effectiveness of CBI as an instructional methodology have been confirmed with adults above 
college age. Thus, while the literature suggests principles of effective CBI, a cogent question remains 
to be answered: To what extent are these principles applicable to adults in general and, specifically, to 
adults in a workplace setting? 

Acccxi^anying the inoeasing pq>ularity and spplicaticHi of CBI in human resource development 
(HRD) contexts must come a clear xmderstanding of whether or not the design principles we are relying 
upon are solidly groxmded. The purpose of this paper therefore is to survey the research literature to 
identify the priiKnples of effective CBI ; assess the adequacy of this research in terms of adult populations, 
particularly adults in the workplace; and make recommendations for a research agenda as related to CBI, 
adults, and the workplace. 

General Research Findings 

Learner Control of CBI A major advantage of CBI is seen in its ability to adapt to the 
individual needs and abilities of learners. One significant aspect of this adaptability is the potential of CBI 
to offer learners differing degrees of control over lesson pace, sequence, content, or quality of instruction; 
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amount of lesson practice or review; the fonn, frequency, or content of feedback; when to test; when 
to terminate instruction; or some combination of these variables. 

The issue of learner ccmtiol is seen by some as the most important design issue in CBI ( Jonassen 
& Hannam, 1 987). Providing such control allows the learner to guide the response of the instructional 
program to suit that individual's needs as he or she perceives them (Williams, 1 993; Taylor, 1 987). 

The underlying assumption of learner control is that the learners themselves are best able to determine 
the type of instruction needed for effective learning. Providing for some degree of learner control 
therefore leads to enhanced learning perfonnance. 

However, the research results regarding the impact of learner control on learning performance 
in CBI are mixed. Some authors have found performance in terms of learning and retention to be 
significantly higher under learner control, as compared to more passive, program control (Ellerman & 
Free, 1990; Carrier & Williams, 1 988). Others (Dalton, 1 990; Kinzie & Sullivan; 1 989) have found no 
difference in learning performance between learner control and program control. Still others (Pollock 
& Sullivan, 1990; Lee & Wong, 1 989) have found learner performance under various forms of program 
control to be sixpenor to that under learner control. There is some evidence that the equivocality of these 
findings may be dependent on a number of individual learner characteristics such as prior knowledge 
(Shaw, 1992), cognitive ability (Kinzie, Sullivan, & Berdel 1988), motivation (Morrison, Ross, & 
Baldwin, 1 992), cognitive style (Carrier, Davidscm, Higscm, & Williams, 1 984), or personality type (Kern 
&Matta,1988). 

Group Methods in CBI The potratial that CBI offers for one-on-one interaction (one learner- 
one computer) has been forwarded as a major advantage of this instructional technology. This view 
asserts that learning is best in CBI when there is one student woiidng with one computer because 
individual learning needs can be identified and addressed as well as individualized feedback and 
reiirforcement developed and administered (Williams, 1993). However, some current research challenges 
this perspective. Webb (1987), for example, reviewed 14 studies comparing group and individual 
methods of instruction in CBI and found no study in which significantly greater learning occurred when 
students worked alone. Stephenson (1992), studying college business students in a spreadsheet CBI 
tutorial, found that students in dyads outperformed all individualized conditions. Dalton, Haimafin, and 
Hoqjer (1989) found that learners working in pairs significantly outperformed individuals in con^uter- 
aided instructional lesson. Johnson, Johnson, and Stanne (1 985) found learners working cooperatively 
in groins of four produced a greater quantity and quality of daily work and demonstrated greater problem- 
solving dolls than learners woiidng individually or competitively. This and other research suggests that 
the nature of the interpersonal interaction present when groups of individuals work cooperatively is an 
important component in the effectiveness of CBI. Peer interaction, for instance, has been found to 
encourage high level elaboration (i.e., e)q)laining) between students which leads to greater learning gains 
(Webb, Ender, & Lewis, 1 986). Students working together in CBI have also been found to be able to 
produce ideas that individuals are unable to generate on their own (Webb, 1 987) and to learn jfrom others 
by observing and imitating the methodologies other students use in problem solving or tadc completion 
(Hythecker, Rocklin, Dansereau, Lambiotte, Larson, & O'Donnell, 1985). 

Feedback in CBI Feedbadc is connncmly defined as information provided to learners following 
a response that is intended to help the learner locate errors and correct mistakes (Schimmel, 1988). 
Feedback has been earmarked as one of the critical variables in the design of effective CBI because of 
the computer's capability to consistently provide a wide range of feedback information (Clariana, Ross, 
& Morrison, 1991). The limited research conducted on feedback in CBI focuses on performance 
improvement and conparisons of different feedback techniques. These techniques vary along a number 
of qualitative dimensions including content, fi^quency, and timing. 

Feedback Content. Feedback content refers to the information provided to learners about the 
appropriateness of their response relative to the instructional content. The infonnaticmal content of 
feedback is generally thought of as lying along a continuum of explanatory complexity fiom single 
confinnation ("correct" or "try again") to various types of explanatory feedback (Schimmel, 1988). 
Several studies have compared the effects of various forms of feedback content on learning (Clariana, 
1993a; Kim &PhiUips, 1991). This research supports a hierarchy of feedback content which says that 
no feedback is less effective in terms of improving performance than confirmatory feedback which is less 
effective than explanatory feedback Other evidence suggests that this hierarchy is not so well established 
and that the provision to leamCTS of extensive information about why their answers are wrong is no more 
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effective than offering minimal information (Schimmel, 1 983). The overall implication of these findings 
is that the optimum level of feedback information may depend on the instructional setting, instructional 
content and objectives, and the individual needs of the learners. 

Feedback Timing. Feedback timing has to do with vv^en in the process of instruction feedback 
is delivered. Feedback may be provided either immediately after a learner's response or delayed for a 
period of time or number of responses (Clariana, et al., 1991 ). The available research addre^ing the 
effect of feedback timing in CBI is limited and the results of what has been done are mixed. Some 
research points "toward small to moderate positive effects of immediate feedback' in the acquisition of 
behavior (Kulik & Kulik, 1 988, p. 93). Other studies have demonstrated that the delay of feedback 
facilitates both acquisition of knowledge and the retention of learning (Gaynor, 1981; Sturges, 1 978). 
Thus, vihereas immediate feedback remains the most common feedback timing convention in CBI there 
is evidence that some delay of feedback may be siq>eriQr under certain conditions. 

Feedback Frequency. Frequency of feedback in CBI deals with the amount of information 
supplied by feedback as well as the technical possibility of applying feedback once (for example, after 
the first response), a second time consequent to a second try by the learner, or to allow the learner to 
continue making attempts followed by feedback until a correct answer is obtained (Clariana, 1990). 
Some writers have suggested that a multiple-try or answer-until-correct feedback strategy may improve 
learning performance to the extent that it engages the learner in additional cognitive processing following 
incorrect responses (Clariana, 1 993b). Only limited research has been aimed at testing whether one or 
many responses with acconpanying feedbadc is mwe effective. The findings of this research indicate that 
there is no significant difference in performance of learners with single-try versus multiple-try feedback. 

Computer Anxiety Computer anxiety refers to a state characterized by a "fear of impending 
interaction with a computer that is disproportionate to the actual threat presented by the computer" 
(Howard, Murphy, & Thomas, 1 987, p. 14). This state involves an array of physiological, attitudirud, and 
emotional responses that may occur vdien individuals interact or expect to interact with computers. 

The current research on con^uler anxiety is inconclusive on two issues. First, to the extent that 
an individual's performance in any given situation is a function of his or her physical, emotional, or 
attitudinal state, canputer anxiety is ctmsidered an important variable in CBI. This presumption is based 
largely on a rather extensive history of research linking high anxiety in general to decreases in 
performance and learning (Tobias, 1979). Although some research indicates that negative emotional 
reactions to computers can have a detrimental impact on the extent to which computers can be effectively 
used (Marcoulides, 1 988) other studies provide contradictory evidence. Bracey (1988), for example, 
found a positive .7 1 correlation between computer anxiety and achievement in a computer information 
systems course. 

A second unresolved issue has to do with the degree to which the rapidly increasing use of 
computer technology has been accompanied by a similar increase in the number of people who feel 
anxious about using computers. Some studies (Elder, Gardner, & Ruth, 1 987; Glass, & idiight, 1 987) 
show that fi*om 30 to over 50% of respondents report attitudes indicative of computer anxiety. Other 
leseaidi (Massoud, 1991; Lewis, 1 988) suggests that these figures overestimate the extent of computer 
anxiety and, moreover, that some populations are relatively unaffected by computer anxiety and have 
fairly positive attitudes towards computers. In sum, although the assumption persists that computer 
anxiety is an important variable in CBI, the research evidence is far &om conclusive in terms of 
identifying or confirming its inq>act on performance as well as its presence across populations and 
settings. 

Screen Design Screen design as a variable in CBI incorpcatites components such as text 
density, screen density, layout and location of mformation, and legibility. The literature in this area is 
largely propositional with only a few studies focused on determining if these components make a 
difference in learning. As with many of the studies done in computer based instruction, most of these 
studies have been done with student populations. 

Text Density. Text density refers to the amount of information presented on the screen. High 
density text is assumed to provide more content-related mformation and more in-depth explanation of the 
information while low density text generally uses a more abbreviated outline format. Two studies 
(Morrison, Ross, O'Dell, & Schultz, 1988; Ross & Morrison, 1988) conducted with undergraduate 
teacher education majors examined the impact of text density on learning. Both of these studies found 
no significant difference in learning attributable to different text density levels. 
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Screen Density. Screen density refers to the proportion of print to white space on a computer 
screen. Scfme research suggests that learner preferences for screen density vary and that screen density 
may impact the efficiency of learning. Morrison, Ross, O’Dell, and Schultz’s (1988) study of 
undergraduates found that learners prefeired moderate white space and did not like the text spread over 
a relatively large number of screens because it forced them to switch screens too often. Other studies 
have estabhshed that reading efficiency in CBI is improved when text is double spaced (Hooper & 
Hannafm, 1 986). No studies to date, however, have estabhshed a relationship between the degree of 
screen density and improved learning performance. 

Location of Information. Maitindale (1993) has argued for text schemas in CBI based on the 
assumption that text structures that are consistent over a range of programs or lessons ease progression 
throu^ a lesson and fadhtate the acquisition of informatioa Hooper and Hannafin (1986) have similarly 
argu^ that the integration of instructional content with consistent schemata and frame design in CBI 
helps to cue q )pn y riate cognitive behavior and promote memory retrieval. The sparse research that has 
ttklressed these issues indicates that the consistent placement of information on a computer screen does 
help in the transfer of information and improvement of learning, suggesting that the location of 
information serves as a mediator of recall and ftmctions as a cue to help organize memory search 
(Aspillaga, 1991). 

Legibility. Legibihty, or the ease of reading material from a computer screen, includes three 
variables - justification, line length, and leading. Justification has to do with line length and the amount 
of ^ace between words. Full justification makes all lines the same length by vaiying the space between 
words; flush left justification has equal spaces between words but produces lines of vaiying length; and 
centered justified equally spaces words and balances the text around an imaginary line down the center 
of the page. The research on justification shows left justified text to be the most efficient convention, 
reading faster than both centered or fiill justified text (Hooper & Hannafin, 1 986). 

Line length deals with the optimum number of characters that can be displayed on a single 
horizontal line. Little or no research has been done in this area of CBI and, although some authors 
suggest that text is read m^e efficiently when presented with greater density (for example, 80 as opposed 
to 40 characters per line), the issue of what line length is best has not been settled. 

Leading refers to the space between lines in computer generated text. As previously noted, 
several studies have reported that reading efficiency is increased when text is double spaced (Hooper & 
Hannafin, 1986). 

Colory Graphics, and Animation. The increased technical capabilities of computers has made 
the incorporation of color, graphics, and animation into CBI a relatively simple task. Although these 
features of CBI can be impressive and easily draw a learner's attention, the question of whether they 
improve learning remains largely unresolved. For example, Baek and Layne (1988) fotind that the 
addition of color, while aesthetically pleasing, did not improve learning in a CBI lesson but that both 
graphics and animation did. Such gains in learning with animation are not, however, consistent across 
studies. Rid:)er (1990) reviewed 1 2 studies using animation and found no difference in learning in CBI 
with animation in 6 of those studies. On the other hand, it was noted that the students in lessons 
enplqying animation \ock significantly less time to answer questions covering lesson content. This may 
indicate that animation aids in the organization of material during the acquisition of information and in 
the reconstruction of material during retrieval. 



CBI Research and Adult Learners 

Learner Control The research on learner control in CBI has shown that such control is not 
uzKonditionally effective as a strategy to improve learning performance and that variotis individual traits 
appear to influence the impact of learner control on performance. This research leaves unresolved the 
more ^^ecific issues of what degree of learner control is best under which circumstances and with which 
learners, particularly adult learners above college age. Most of the studies in this area have been 
ccmducted with college age subjects (x younger. For exan^le, from the 28 studies reviewed for this paper 
only 2 reported results from subjects above undergraduate college age and over half (n=15) were 
conducted with hi^ school age subjects or younger. Of the 2 studies done with adults, Ellerman and Free 
(1990) found hi^er levels of instructional sequence control in paired associate learning to be correlated 
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with significantly better learning performance in subjects whose mean age was 27. Similar results were 
found for a group of students with a mean age of 23. A major conclusion of Shaw (1 992), based on 6 
years of research with computer-aided instruction at the U.S. Army Construction Engineering Research 
Laboratory was that, under CBI, adults are best served with relatively greater degrees of instructional 
control. Shaw summarized her research by stating that CBI "targeted at adult professionals who are 
evaluated cm the basis of job performance and who need to learn while th^ work must be quite different 
from instnicticmal programs for school children, ^^ose objectives are subject matter mastery and whose 
evaluations are test scores" (p. 57). 

In addition,, the contradictory findings with regard to learner control have been interpreted to 
indicate that some learners may lack the metacognitive skills to use instructional control effectively 
O^illiams, 1993). To the extent that adult learners can be presumed to have developed more refined and 
effective cognitive strategies this interpretation suggests that adults may be better able than younger 
subjects to profit fixnn learner control of instruction in CBI. Penland (1 979), for instance, suggests t^t 
adult learners m£^ prefer learner controlled instructicm because they can set their own leaning pace, use 
their own style of learning, keep the learning strategy flexible, and impose their own structure on the 
learning project Again, these and other questions regarding learner controlled CBI with adults in various 
settings have not been empirically tested and remain in large part unanswered. 

Group CBI There is a rich body of research supporting the use of group methods as a means 
to promote effective learning in CBI. The general implication of these findings is that it is possible to 
design effective groi]p-oentered CBI for a wide range of learners. For example, different types of group 
interactions, including receiving explanations, making suggestions, and providing answers to specific 
questions, have been consistently found to be positively correlated with achievement gains among college 
age subjects (Webb, 1987). Whether these types on group interactions will be equally effective with 
learners above college age or adults in the workplace is largely unknown. No work currently in the 
literature has been done to investigate the ways in which the design or function of cooperative learning 
in CBI affects adult learners in different settings. The research thus leaves many questions unanswered 
regarding tte effectiveness (x dynamics of group CBI with adults, the optimum groups size for adults, or 
ways in which software can be designed for effective small group use with adults. 

Feedback in CBI Nearly all of the feedback studies conducted have involved subjects of 
college age or younger leaving unaddressed many questions about the relationship between adult learners 
and the qualitative aspects of feedback in CBI. In general, the research conducted in this area suggests 
that instructional designers should seek "to encourage the mindful engagement most appropriate for the 
individual learner" (Clariana, 1 993b, p. 7 1 ). This declaration, vshile confirming that different learners 
may have different prefoences for different types of feedback, provides litde guidance for the design and 
application of appropriate feedback in different settings with different learners. There are many ways, 
for example, in which adult learners may differ fi^m younger learners in their feedback needs and 
preferences. In terms of feedback fi^quency, it is conceivable that the metacognitive abilities of adults 
allow them to make use of the additional information provided by multiple-try feedback while that same 
fiequoKyoffeedbadc may hinder the performance of less experienced learners. It is similarly possible 
that, to the extent that a differential in tl^ effectiveness of feedback timing exists across populations, such 
a differential correlates with age. Cohen (1985) suggested that the metacognitive abihties of adult 
learners are better suited to delayed feedback, particularly that provided at the end of a lesson, because 
they are better able to use the information at that point in the lesson to summarize and organize 
mfonnationaiKi to assess how they are progressmg The metacognitive abilities of adult learners may also 
oihance the effectiveness ofmcxecon^lex as opposed to simple confirmatory feedback. In short, despite 
its in^licit value, feedback remains aot of the least e>q7lored variables in terms of the nature of its impact 
on adult learning performance in CBI. 

Computer Anxiety There is speculation in the literature about the negative effects computer 
anxiety may have on the self-worth or self-esteem of learners (Morrow, Prell, & McElroy, 1 986), its 
power to alienate workers fixmi their jobs (Elder, et. al., 1 987), or even to lead to acts of subversion in 
the workplace (Baumgarte, 1984). It may be similarly speculated that adults in the workplace will 
exhibit higher levels of computer anxiety to the extent that they perceive computers as a threat to job 
performanoe or job security. Or that adults who view computer expertise as something that will improve 
their job performance will be relatively unaffected by computer anxiety. These views, however, are 
largely unsubstantiated by researdi Indeed, there is little research establishing the incidence of computer 
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anxiety across populations and settings, documenting the effects of computer anxiety on the human- 
conputer intoface, or the specific impact of computer anxiety on adult learning in CBI. More research 
is needed addressing these and other questions so that we may verify whether and to what extent 
ccxi^uter anxiefy threatens productivity and learning in situations in which computers are used for work 
or instruction. 

Screen Design Most of the research done in the area of screen design has been done with 
samples from student populations, btuitively, there is reason to question the applicability of these 
findings to adult lexers because as individuals age, physical changes in eye sight, coordination, and 
reflexes naturally occur. These changes may, in turn, effect the impact of screen design elements on 
learning. Fch* exanq^le, dianges in visual acuity has inq)lications for screen clarity and text legibility (print 
size, color, and so on) and its impact on efficiency of reading the computer screen. Older learners may 
therefore profit instructionally fi-om larger print or more dramatic color distinctions. The cognitive or 
learning style differences of adult learners may have in^>lications for the use of animation. Reibcr ( 1 990), 
for example, cited several studies ccmdiKted with adults and animation in CBI which collectively indicate 
that adults may be better able to visualize information and are therefore less dependent on visual aids 
such as animation (x graphics. Similarly, the experiential or metacognitive differences adult learners may 
indicate a preference or r^ed for less instructional support information during learning. If so, this would 
have implications for schema development in CBI and the amount or location of information presented 
on CBI screens. 



Conclusion 

Cofiq)uter-based instruction is without question an increasingly popular instructional medium for adult 
educators and HRD practitioners. Accompanying the growing use of CBI is a concurrent need for solidly 
grounded design principles to assist us in the use of this technology. This brief survey of the literature 
has undersccml two inajcH'shcxtcomings found in the research on CBI. First, in areas of learner control, 
feedback, and screen design existing studies are either very limited in number or the findings are 
inconsistent and inconclusive. As a result, there are few well established technical or didactic guidelines 
fcx^ designing effective CBI for different content areas, settings, or with different learners. Secondly, the 
findings that have been adequately confinned are difficult to generalize to adults because the research in 
these areas has been dcme almo^ singularly with subjects of college age or younger. For example, of the 
5 1 4 studies reviewed directly and in literature reviews for this paper, 467 were done with subjects of 
college age or younger while only 47 were done with adults. This emphasis is understandable to the 
extent that students represent a readily available and inexpensive population fix>m which to draw the 
necessary research samples. Althou^ the findings fi-om studies employing these subjects may not be 
inaccurate, generalizing these results to adults is not readily justifiable either in terms of relevance to 
workplace problems or in terms of sample representativeness. The combined effect of these two 
shortccxnings, limited researdi -based guidelines and findings suspect in terms of generalizability, raises 
questions about the usefulness and acceptability of ciirrent CBI design principles for adults in the 
workplace. Research examining the fipplication of CBI variables in organizational settings with non- 
student populations is needed to extend the generalizability of current findings. 
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